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DETAILED ACTION 
Specification 

1 . Applicant is reminded of the proper content of an abstract of the disclosure. 

A patent abstract is a concise statement of the technical disclosure of the patent and 
should include that which is new in the art to which the invention pertains. If the patent is of a 
basic nature, the entire technical disclosure may be new in the art, and the abstract should be 
directed to the entire disclosure. If the patent is in the nature of an improvement in an old 
apparatus, process, product, or composition, the abstract should include the technical disclosure 
of the improvement. In certain patents, particularly those for compounds and compositions, 
wherein the process for making and/or the use thereof are not obvious, the abstract should set 
forth a process for making and/or use thereof. If the new technical disclosure involves 
modifications or alternatives, the abstract should mention by way of example the preferred 
modification or alternative. 

The abstract should not refer to purported merits or speculative applications of the 
invention and should not compare the invention with the prior art. 

Where applicable, the abstract should include the following: 

(1) if a machine or apparatus, its organization and operation; 

(2) if an article, its method of making; 

(3) if a chemical compound, its identity and use; 

(4) if a mixture, its ingredients; 

(5) if a process, the steps. 

Extensive mechanical and design details of apparatus should not be given. 

2. The abstract of the disclosure is objected to because the abstract refers to purported 
merits: "...synthesized images can be displayed with less work and time." Correction is 
required. See MPEP § 608.01(b). 

3. The title of the invention is not descriptive. A new title is required that is clearly 
indicative of the invention to which the claims are directed. 

The following title is suggested: Method and apparatus for annotating panoramic image- 



based walkthroughs. 
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Drawings 

4. The drawings are objected to as failing to comply with 37 CFR 1 ,84(p)(5) because they 
include the following reference character(s) not mentioned in the description: C4 (FIG. 13). 
Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the 
specification to add the reference character(s) in the description in compliance with 37 CFR 
1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any 
amended replacement drawing sheet should include all of the figures appearing on the immediate 
prior version of the sheet, even if only one figure is being amended. Each drawing sheet 
submitted after the filing date of an application must be labeled in the top margin as either 
"Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If the changes are not 
accepted by the examiner, the applicant will be notified and informed of any required corrective 
action in the next Office action. The objection to the drawings will not be held in abeyance. 



Claim Rejections - 35 USC §101 

5. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or 
any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 

6. Claims 1-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed 
to non-statutory subject matter. 

7. Claim 6 and 1 1 recites a computer program not technologically embodied to enable the 
functionality to be realized. Computer programs, per se, are not in one of the statutory categories 
of invention. Functional descriptive material claimed in combination with an appropriate 
computer readable medium to enable the functionality to be realized is patent eligible subject 
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matter if it is capable of producing a useful, concrete and tangible result when used in the 
computer system. See MPEP § 2106 with regard to computer programs. 

8. Claims 1-5 and 7-10 appear to be to an abstract idea rather than a practical application of 
the idea. Claims 1-5 and 7-10 does not result in a physical transformation nor does it appear to 
provide a useful, concrete and tangible result. Specifically, it does not appear to produce a 
tangible result because merely computing the positions of annotations and synthesizing images 
are nothing more than thoughts or computations within a processor. It fails to use or make 
available for use the result of image synthesis to enable its functionality and usefulness to be 
realized. Additionally, the asserted practical application in the specification of information 
processing is reproducing the synthesized image on a display unit. The practical application is 
not explicitly recited in the claims nor does it flow inherently therefrom. 

9. Claim 12 preempts an abstract idea. A claim may not preempt every substantial practical 
application of an abstract idea, law of nature or natural phenomena because it would in practical 
effect be a patent on the judicial exceptions itself. 

10. To expedite a complete examination of the instant application, the claims rejected under 
35 U.S.C. 101 as non-statutory subject matter are further rejected as set forth below in 
anticipation of applicant amending the claims to place them within the four categories of 
invention. 

Claim Rejections - 35 USC § 102 

1 1 . The following is a quotation of the appropriate paragraphs of 35 U.S.C. 1 02 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 



Application/Control Number: 10/763,222 Page 5 

Art Unit: 2628 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

12. Claims 1, 3, 4, 6, 7, 11, and 12 are rejected under 35 U.S.C. 102(b) as being 
anticipated by Masakatsu Kourogi, Takeshi Kurata, Katsuhiko Sakaue, and Yoichi 
Muraoka, "A panorama-based technique for annotation overlay and its real time 
implementation/' July 30, 2000, Proceedings of IEEE International Conference on 
Multimedia and Expo, p. 657-660 (herein referred to as "Kourogi et al"). 

13. With regard to claim 1, Kourogi et al discloses "an information processing method 
comprising: 

a. a viewpoint position/sight line direction determination step of determining a 
viewpoint position and a sight line direction on a map {2 nd paragraph of section 2: "The 
referred panoramic image will be switched if necessary as the camera moves around... By 
tracking which panorama is referred, we can also estimate the position and trajectory of 
the user. "; Figure 8 shows an Estimation of the user 's position; Figure 9 shows 
Estimation of the user 's orientation; Figure 2 shows a map); 

b. an annotation display position determination step of determining an annotation 
display position of an object (Abstract: "It finds image alignment parameters between an 
input frame and the panoramic image and then maps the positions of annotations from 
the panoramic image to the input frame and displays the input frame overlaid with those 
annotations. "), from the position of said object on the map determined based on 
observation directions of said object in plural panoramic images, the viewpoint position, 
and the sight line direction (Figure 2 shows "the panoramic image used is switched while 
the user moves "); and 
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c. a synthesis step of synthesizing an annotation image to the annotation display 
position on an actually taken image corresponding to the viewpoint position" (Figure 7 
shows actually taken images overlaid with annotations; 1 st paragraph of section 3.3: "In 
the experiments the user moved along the path A ->B —>C —>D (Figure 5). The output 
video frames overlaid with annotations are shown in Figure 7, and the user *s estimated 
position and orientation are shown in Figure 8 and 9 "). 

14. With regard to claim 3, Kourogi et al discloses "said annotation display position 
determination step determines the annotation display position of the panoramic image located 
between said plural panoramic images, by using the determined position of the object on the 
map" (2 nd paragraph of section 2: "The referred panoramic image will be switched if necessary 
as the camera moves around. The neighborhood relationship between panoramas are given to 
enable the switching to occur. "; Figure 2 shows annotations for Panoramic image at point B, 
between panoramic images at point A and C ). One of ordinary skill in the art would recognize 
the neighborhood relationships are analogous to the position of the object on the map from 
Figures 2 and 5. 

15. With regard to claim 4, Kourogi et al discloses "the determined annotation display 
position can be manually adjusted" (1 st paragraph of section 2: "Our approach uses a panoramic 
image to which annotations are manually attached as the source of information. "). 

16. Claim 6 recites the limitations of claim 1 as a "control program for causing a computer to 
execute a information processing method." Kourgoi et al shows the limitations recited in claim 

1 , and a "control program for causing a computer to execute a information processing method" 
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{section 3: "We implemented our method as software running on a PC cluster and evaluated it in 
experiments... "). 

17. With regard to claim 7, Kourogi et al discloses "an information processing method, used 
in an image reproduction apparatus for achieving walk-through in a virtual space represented by 
using an actually taken image, of synthesizing an annotation image to the actually taken image, 
said method comprising the steps of: 

d. setting an annotation display position in each of the plural actually taken images 
{2 nd paragraph of section 1: 'The proposed method uses (1) a set of panoramic images 
acquired at various points in the environment, (2) annotations attached to the panoramas 
and (3) neighborhood relationships between panoramas as prior knowledge about the 
environment. "); 

e. calculating an annotation display position to another actually taken image located 
between the plural actually taken images, by using the annotation display positions 
respectively set in the plural actually taken images {1 st paragraph of section 2: "Our 
approach uses a panoramic image to which annotations are manually attached as the 
source of information. When an input frame is given, it is aligned with the referred 
panoramic image. Then, we can map the positions of annotations from the panorama to 
the frame... "); and 

f. synthesizing the annotation image to the actually taken image on the basis of the 
calculated annotation display position {1 st paragraph of section 2: " Then, we can map 
the positions of annotations from the panorama to the frame, and thus can create the 
frame overlaid with the annotations as shown in Figure 1. "). 
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1 8. Claim 1 1 recites the limitations of claim 7 as a "control program for causing a computer 
to execute a information processing method." Kourgoi et al shows the limitations recited in 
claim 7, and a "control program for causing a computer to execute a information processing 
method" (section 3: "We implemented our method as software running on a PC cluster and 
evaluated it in experiments... "). 

19. Claim 12 recites the limitations of claim 1 as a "an image reproduction apparatus." 
Kourgoi et al shows the limitations recited in claim 1 , and "an image reproduction apparatus" 
(section 3: "We implemented our method as software running on a PC cluster and evaluated it in 
experiments which a user was equipped with a wearable display, a wearable camera, and a 
wearable computer. "). 

Claim Rejections - 35 USC § 103 

20. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 



21. The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 
(1966), that are applied for establishing a background for determining obviousness under 35 
U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating obviousness 
or nonobviousness. 
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22. This application currently names joint inventors. In considering patentability of the 
claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various 
claims was commonly owned at the time any inventions covered therein were made absent any 
evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1 .56 to point out 
the inventor and invention dates of each claim that was not commonly owned at the time a later 
invention was made in order for the examiner to consider the applicability of 35 U.S.C. 103(c) 
and potential 35 U.S.C. 102(e), (f) or (g) prior art under 35 U.S.C. 103(a). 

23. Claim 2 is rejected under 35 U.S.C. 103(a) as being unpatentable over Kourogi et al 
in view of U.S. Patent No. 6,563,529 to Jongerius. 

24. With regard to claim 2, Kourogi et al discloses "a two-dimensional map" in Figure 2 and 
Figure 5. Kourogi et al does not disclose the map is an image. Jongerius discloses a panoramic 
image from a viewpoint on a given map displayed along with an image of the two-dimensional 
map image (Figure 4\ lines I 7-22 of column 5: "As will be explained, detailed field of view 40 
shows a new area of the panoramic view of FIG I slightly to the left of the view of FIG. 2, while 
map 38 shows a new highlighted view area 36, rotated slightly counter-clockwise (CCW) from 
view area 34 of FIG. 2. "). 

25. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to incorporate a two-dimensional map image as disclosed by Jongerius in the method and 
system disclosed by Kourogi et al. The motivation for doing so would have been to provide the 
user with a better understanding of the area being viewed. Therefore, it would have been obvious 
to combine Kourogi et al with Jongerius to obtain the invention specified in claim 2. 
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26. Claim 5 is rejected under 35 U.S.C. 103(a) as being unpatentable over Kourogi et al 
in view of U.S. Patent No. 6,563,529 to Jongerius and in further view of U.S. Patent No. 
6,392,658 to Oura. 

27. With regard to claim 5, Kourogi et al discloses the limitations recited in parent claim 1 . 
Furthermore, Kourogi et al discloses determining an "observation direction of the object" in 
Figures 8 and 9 and a two-dimensional map. However, Kourogi et al does not disclose "a 
graphical user interface including a map display portion." Jongerius discloses "a graphical user 
interface including a map display portion and a panoramic image display portion is provided 
(Figure 4\ lines 1 7-22 of column 5: 11 As will be explained, detailed field of view 40 shows a new 
area of the panoramic view of FIG. 1 slightly to the left of the view of FIG. 2, while map 38 
shows a new highlighted view area 36, rotated slightly counter-clockwise (CCW) from view area 
34 of FIG. 2. "), said plural panoramic images are selected by using the map display portion {lines 
46-47 of column 2: "...any change in the map image is immediately reflected in the detailed 
image. "). 

28. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to incorporate a two-dimensional map image as disclosed by Jongerius in the method and 
system disclosed by Kourogi et al. The motivation for doing so would have been to provide the 
user with a better understanding of the area being viewed. 

29. Neither Kourgoi et al nor Jonerius disclose "the observation direction of the object is 
designated on the selected panoramic image displayed on the panoramic image display portion." 

30. Oura discloses "the observation direction of the object is designated on the selected 
panoramic image displayed on the panoramic image display portion" (Figure 6 shows two views 
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of the panoramic image: the top image uses a frame to indicate the observation direction of the 
viewer depicted in the lower images; lines 21-23 of column 5: "A part of the whole picture is 
framed. The framed region is enlarged and displayed as a partial picture. "). 

3 1 . At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to designate the observation direction on the panoramic image display portion as 
suggested by Oura in the method and system disclosed by the combination of Kourogi et al and 
Jongerius. The motivation for doing so would have been to provide the user with a better 
understanding of the location of the current view within the available panoramic image data. 
Therefore, it would have been obvious to combine Kourogi et al and Jongerius with Oura to 
obtain the invention specified in claim 5. 

32. Claim 8 is rejected under 35 U.S.C. 103(a) as being unpatentable over Kourogi et al 
in view of Reid Harmon, Walter Patterson, William Ribarsky and Jay Bolter, "The Virtual 
Annotation System," March 30, 1996, Proceedings of the 1996 Virtual Reality Annual 
International Symposium, p. 239-245 (herein referred to as "Harmon"), 

33. . With regard to claim 8, Kourogi et al discloses "the setting of the annotation display 
position in each of the plural actually taken images is performed according to a user's manual 
instruction" (1 st paragraph of section 2: "Our approach uses a panoramic image to which 
annotations are manually attached as the source of information. "). Kourogi et al does not 
expressly disclose "and the calculated annotation display position can be adjusted based on a 
user's manual instruction." However, Harmon discloses "the calculated annotation display 
position can be adjusted based on a user's manual instruction" (2 nd paragraph of section "Our 
annotation system ": 'When an annotation is made for an object, an annotation icon is created 
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and placed in the VE, and a line connects the icon to the object. ..If an icon of this type is moved, 
the system saves its new position and adjusts the line to the annotated object accordingly. "). 

34. Kourogi et al and Harmon are analogous art because they are from the same field of 
endeavor and problem solving area: annotating virtual environments. At the time of the 
invention, it would have been obvious to a person of ordinary skill in the art to allow the user to 
adjust annotations as taught by Harmon in the system disclosed by Kourogi et al. The motivation 
to do so would have been to resolve the problem that occurs when the calculated position of the 
annotation occludes objects that the user desires to view. Therefore, it would have been obvious 
to combine Kourogi et al with Harmon to obtain the invention specified in claim 8. 

35. Claims 9 and 10 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Kourogi et al in view of Erwin Pang, and Dimitrios Hatzinakos, "An Efficient 
Implementation of Affine Transformation Using One-Dimensional FFT's," April 1997, 
Proceedings of 1997 IEEE International Conference on Acoustics, Speech, and Signal 
Processing, Vol. 4, p. 2885-2888 (herein referred to as "Pang et al"). 

36. With regard to claims 9 and 10, Kourogi et al discloses in the first paragraph of section 
2.1: 

We estimate image alignment parameters between an input frame and a referred 
panoramic image, by using a fast and robust gradient-based method that can find affine 
or projective parameters of image alignment between images. We use affine model 
because of its stability of estimation. 

37. Kourogi et al does not expressly disclose "performing interpolation to the annotation 
display position set in each of the plural actually taken images," or "the interpolation is non- 
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linear interpolation" and "from among plural non-linear curves previously held, the non-linear 
curve is determined based on the annotation position of the object in each of the plural actually 
taken images," as recited in claims 9 and 10. Pang et al teaches in the first paragraph of section 1 
{emphasis added): 

Geometrical transformation of digital image is a common application in image 
processing [I] [2]. The uniformly distributed samples on a two-dimensional plane, after 
being transformed, will be misaligned with the reference grid pattern which are only 
defined for discrete locations. An interpolation of these transformed pixels is thus needed 
to recover those on the grid points. 

38. Pang et al further discloses, "Cubic interpolation [1 1] is more accurate since it uses more 
neighbouring points for resampling" {1 st paragraph of section 4.1). 

39. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to incorporate non-linear interpolation for determining the annotation position as taught 
by Pang et al in the system disclosed by Kourogi et al, and to do so from among plural non-linear 
curves previously held. The motivation for doing so would have been to "recover the grid points 
in the image" after the affine transformation is applied, as suggested by Pang et al, in a 
computationally efficient manner. Therefore, it would have been obvious to combine Pang et al 
with Kourogi et al to obtain the invention specified in claims 9 and 10. 

Conclusion 

40. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. Ismo Rakkolainen, Jani Timmerheid, Teija Vainio, "A 3D City Info for Mobile 
Users," November 9, 2000, Proceedings of the 3rd International Workshop on Intelligent 
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Interactive Assistance and Mobile Multi-Media Computing, p. 115-121 shows a three- 
dimensional view and a corresponding map view. U.S. Patent No. 6,968,973 to Uyttendaele et al 
discloses an annotated image-based virtual tour. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jason M. Repko whose telephone number is 571-272-8624. The 
examiner can normally be reached on Monday through Friday 8:30 am -5:00 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka Chauhan can be reached on 571-272-7782. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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